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Abstract —  In  this  paper,  we  design  an  adaptive  kinematic 
controller  that  asymptotically  regulates  a  robot  end-effector 
to  a  desired  position  and  orientation  under  visual  feedback 
(of  points  located  on  a  fixed  reference  frame)  from  a  cam¬ 
era  mounted  on  the  end-effector.  This  task  is  accomplished 
despite  lack  of  depth  measurements  as  well  as  uncertainty 
in  the  camera  intrinsic  parameter  matrix. 


I.  Introduction 

During  the  last  5  years,  2.5D  visual  servo  controllers 
have  become  very  popular  given  their  amenability  to  con¬ 
trol  analysis  as  well  as  the  robustness  that  these  approaches 
bring  to  robot  end-effector  tracking  and  regulation  applica¬ 
tions.  Roughly  speaking,  2.5D  visual  servoing  involves  ob¬ 
taining  information  from  the  3D  task-space  (either  through 
a  given  3D  model  or  more  interestingly  through  a  projec¬ 
tive  Euclidean  reconstruction)  to  regulate  the  rotation  and 
depth-related  error  systems  while  simultaneously  utilizing 
information  from  the  2D  image-space  to  regulate  the  pla¬ 
nar  translation  related  error  systems.  Drawing  upon  the 
aforementioned  novel  idea  of  fusing  2D  and  3D  informa¬ 
tion,  various  kinematic  control  strategies  were  designed  by 
Malis  and  Chaumette  [1],  [2],  [14],  and  [15]  for  the  camera- 
in-liand  problem.  More  recently,  Corke  and  Hutchinson  [5] 
developed  a  new  hybrid  image-based  visual  servoing  scheme 
in  order  to  decouple  rotation  and  translation  components 
about  the  z-axis  from  the  remaining  degrees  of  freedom  so 
as  to  address  the  problem  of  desirable  image-space  trajec¬ 
tories  resulting  in  undesirable  Cartesian  trajectories.  Most 
of  the  above  controllers  conjecture  that  a  constant,  best- 
guess  estimate  of  the  depth  information  can  be  utilized  in 
lieu  of  the  exact  value.  Motivated  by  the  desire  to  actively 
compensate  for  unmeasurable  depth  information,  Conti- 
celli  developed  an  adaptive  kinematic  controller  in  [4]  to 
ensure  uniformly  ultimately  bounded  set-point  regulation 
provided  conditions  on  the  translational  velocity  and  the 
bounds  on  uncertain  depth  parameters  are  satisfied.  In  [7], 
Fang  et  al.  recently  developed  a  2.5D  visual  servo  controller 
to  asymptotically  regulate  a  manipulator  end-effector  by 
developing  an  adaptive  update  law  that  actively  compen¬ 
sates  for  an  unknown  depth  parameter.  In  [8],  Fang  et 
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al.  also  developed  a  camera-in-hand  regulation  controller 
that  incorporated  a  nonlinear  robust  control  structure  to 
compensate  for  uncertainty  in  the  extrinsic  calibration  pa¬ 
rameters. 

Most  extant  monocular  2.5D  visual  servoing  controllers 
rely  upon  knowledge  of  the  intrinsic  camera  calibration  pa¬ 
rameters.  In  a  few  previous  results  (e.g.,  see  [8],  [15]), 
nominal,  best-guess  parameters  are  employed.  To  further 
engage  and  understand  robustness  issues,  this  paper  ad¬ 
dresses  the  twin  problems  of  uncertainty  in  the  intrinsic 
camera  calibration  parameters  and  lack  of  depth  measure¬ 
ments.  Specifically,  by  exploiting  the  triangular  structure 
of  the  calibration  matrix,  we  design  an  adaptive  control 
strategy  via  a  Lyapunov-based  approach.  The  control 
problem  is  challenging  because  the  unknown  camera  cal¬ 
ibration  matrix  forces  us  to  work  with  the  pixel  hornog- 
raphy  matrix  instead  of  the  more  facile  Euclidean  coor¬ 
dinates.  The  problem  is  further  complicated  by  the  fact 
that  the  projected  pixel  homography  has  more  unknowns 
than  independent  equations;  hence,  it  is  impossible  to  sep¬ 
arate  the  rotational  and  translational  degrees  of  freedom. 
Fortunately,  the  use  of  the  plane  at  infinity  (i.e.,  the  van¬ 
ishing  points)  provides  additional  information  that  can  be 
used  to  surmount  this  issue.  Given  this  isolation  of  the 
rotation-like  information  that  is  provided  by  the  vanishing 
points,  it  becomes  possible  to  embed  the  intrinsic  parame¬ 
ter  uncertainty  in  the  kinematic  model  to  facilitate  adap¬ 
tive  control  techniques  for  a  special  class  of  multiple-input, 
multiple-output  systems.  To  the  best  of  our  knowledge, 
this  is  the  first  result  that  regulates  the  robot  end-effector 
to  a  desired  position/orientation  through  visual  servoing 
by  actively  compensating  for  the  lack  of  depth  measure¬ 
ments  and  uncertainty  in  the  camera  intrinsic  calibration 
matrix  with  regard  to  the  6  degrees-of-freedom  regulation 
problem. 

This  paper  is  organized  in  the  following  manner.  In  Sec¬ 
tion  II,  a  geometric  relationship  is  expressed  between  the 
images  of  interest  in  terms  of  the  orientation  and  position 
of  the  camera  located  on  the  robot  end-effector.  In  Section 
III,  we  describe  extraction  procedures  for  various  signals 
embedded  in  the  pixel  information  that  facilitate  the  con¬ 
trol  design.  In  Section  IV,  the  control  objective  is  devel¬ 
oped  along  with  the  open-loop  error  dynamics.  In  Section 
V,  a  kinematic  controller  is  developed  via  a  Lyapunov  based 
approach.  Concluding  remarks  are  given  in  Section  VI. 
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II.  Geometric  Model 


III.  From  Geometry  to  Control 


Consider  a  reference  plane  7r  located  on  an  object  that 
has  four  target  points1  (no  three  of  which  are  collinear) 
that  are  denoted  by  <3;  Vi  =  1,  2,  3,  4.  Consider  two  or¬ 
thogonal  coordinate  systems,  denoted  by  T  and  T* ,  where 
T  is  attached  to  a  camera  that  is  held  by  the  robot  end- 
effector,  and  T*  is  a  fixed  coordinate  system  that  represents 
the  constant,  desired  camera  position  and  orientation.  The 
Euclidean  coordinates  of  the  target  points  on  7r  can  be  ex¬ 
pressed  in  terms  of  T  and  T* ,  respectively,  as  follows 

m{t)  =  [  Xi{t)  yi(t)  Zi(t)  ]T  , 

TO*  A  [  X*  y*  Z*  ]T 

under  the  standard  assumption  that  the  distances  from  the 
origin  of  the  respective  coordinate  frames  to  tx  along  the 
focal  axis  remain  positive  (i.e.,  Zi  (t),  z*  >  e  >  0  where  e  is 
an  arbitrarily  small  positive  constant).  The  rotation  from 
T  to  T*  is  denoted  by  R(t )  £  SO( 3),  and  the  translation 
from  T  to  T*  is  denoted  by  Xf  (t)  £  5R3  where  Xf  ( t )  is 
expressed  in  T.  From  the  geometry  between  the  coordinate 
frames  depicted  in  Fig.  1,  to*  can  be  related  to  frii(t)  as 
follows 

fhi=Rm*+Xf.  (2) 

Also,  from  the  geometry  given  in  Fig.  1,  the  distance  d*  £ 
5R  from  T*  to  7r  along  the  unit  normal  is  given  by 


d*  =  n*Tm*  (3) 

where  n*  £  5R3  denotes  the  constant  unit  normal  to  n  ex¬ 
pressed  in  T* .  From  (3),  the  relationships  in  (2)  can  be 
expressed  as  follows 


rrii  =  (R  +  Xh,n*T)  to* 


(4) 


where  Xh  ( t )  £  5R3  denotes  a  scaled  translation  vector  that 
is  defined  as  follows 


(5) 


The  relationship  given  in  (4)  is  a  conduit  to  obtaining  the 
translational  and  rotational  error  between  T  and  T* .  How¬ 
ever,  the  Euclidean  coordinates  of  the  points  to*  (t)  and  to* 
cannot  be  directly  measured;  therefore,  we  must  utilize  the 
pixel  information  gathered  by  the  camera  to  develop  a  rela¬ 
tionship  between  the  current  and  desired  pixel  coordinates 
of  the  points  Oi  in  the  reference  plane  n.  If  the  intrinsic 
camera  calibration  parameters  were  known,  standard  re¬ 
construction  schemes  could  then  be  applied  to  obtain  mea¬ 
surable  translation  and  rotation  error  signals  from  which 
point  a  control  algorithm  could  be  developed  to  regulate 
T  to  T* .  However,  we  are  interested  in  developing  a  con¬ 
trol  strategy  with  the  additional  constraint  of  uncertainty 
with  regard  to  the  intrinsic  camera  calibration  parameters. 
In  light  of  this  uncertainty,  the  best  that  we  can  hope  to 
do  is  to  obtain  measurable  signals  that  mimic  the  rotation 
and  translation  between  T  and  T *  from  a  decomposition  of 
the  pixel  homography,  and  thereby,  facilitate  the  design  of 
kinematic  control  inputs  that  provide  for  asymptotic  reg¬ 
ulation  of  the  robot  end-effector.  This  section  aims  to  set 
up  the  requisite  signals  that  the  subsequent  control  design 
will  utilize. 

A.  Euclidean  and  Projective  Homographies 

We  begin  by  defining  the  normalized  Euclidean  coordi¬ 
nates  TO*  (f),  to*  £  5R3  of  Oi  expressed  in  terms  of  T  and 
T*  ,  respectively,  as  follows 


(6) 

(7) 


The  rotation  and  translation  between  the  coordinate  sys¬ 
tems  can  now  be  related  in  terms  of  the  normalized  coor¬ 
dinates  as  follows 


(8) 


where  cti  (t)  £  5R  denotes  the  depth  ratios,  and  H  ( t )  £ 
?R3x3  denotes  a  Euclidean  homography  [10]. 

Each  target  point  Oi  will  have  a  projected  pixel  coordi¬ 
nate  expressed  in  terms  of  T  (denoted  by  (t),  Vi  ( t )  £  5ft) 
and  T*  (denoted  by  u*,  v*  £  5ft)  that  are  defined  as  ele¬ 
ments  of  pi  (t)  (actual  time-varying  image  points)  and  p* 
(constant  reference  image  points),  respectively,  as  follows 


Pi=[Ui  Vi  1  ]T  p*  =  [  u*  v*  1  ]T  •  (9) 


Fig.  1.  Motion  and  structure  parameters. 


1It  should  be  noted  that  if  four  coplanar  target  points  are  not  avail¬ 
able  then  the  subsequent  development  can  exploit  the  classic  eight- 
points  algorithm  [14]  with  no  four  of  the  eight  target  points  being 
coplanar. 


To  calculate  the  Euclidean  homography  given  in  (8)  from 
pixel  information,  the  projected  2D  pixel  coordinates  of  the 
target  points  are  related  to  to*  (i)  and  to*  by  the  following 
pin-hole  lens  model 

Pi  =  Aim  P*  =  Am*  (10) 
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where  A  £  5R3x3  is  an  unknown,  constant,  and  invertible  in¬ 
trinsic  camera  calibration  matrix  that  is  defined  as  follows 


[14] 


ai  a  2  C14 

A  =  0  0,3  0,5 

0  0  1 


(ID 


where  a*  Vi  =  1,  2, 5  are  scalar  constants,  and  a±,  0,3  >  0. 
After  substituting  (8)  into  (10),  the  following  relationship 
can  be  developed 


where  G  (t)  =  [gij(t)]  Vi,j  =  1,2,3  £  JR3x3  denotes  a 
projective  homography  [10]. 


B.  Projective  Homography  Decomposition 
From  (8)  and  (12),  we  can  rewrite  G  (t)  as  follows 

G  =  R  +  xhn*T  (13) 

where  the  signals  R(t)  £  JR3x3  and  Xh(t),h*  £  5R3  are 
defined  as  follows 


m* .  The  rotational  control  objective  is  complicated  by  the 
fact  that  R  ( t )  is  not  a  measurable  signal;  however,  the 
surrogate  signal  R(t)  of  (14)  can  be  utilized  along  with 
an  adaptive  algorithm  to  compensate  for  uncertainty  in 
the  calibration  matrix  A.  From  the  second  relationship  of 
(14)  and  the  full-rank  nature  of  A,  the  translational  con¬ 
trol  objective  can  be  achieved  by  regulating  Xh  (t)  to  zero. 
However,  Xh  ( t )  is  only  available  up  to  a  scale  factor  that 
has  an  indeterminate  sign  (details  available  upon  request). 
Moreover,  it  is  desirable  from  a  robustness  point  of  view  to 
servo  on  actual  pixel  information  (in  lieu  of  reconstructed 
information)  to  increase  the  likelihood  that  the  object  will 
stay  in  the  field  of  view  of  the  camera.  Therefore,  we  state 
the  translational  control  objective  as2:  pi  ( t )  — >  p\  and 
a%  ( t )  — ►  1.  From  the  definitions  of  (8)  and  (10),  we  obtain 
rhi  ( t )  — >  m\.  From  the  definition  of  (4),  (5),  and  the  fact 
that  the  control  will  ensure  that  R  ( t )  — >  I3 ,  we  can  see 
that  Xf  (t)  —>  0  This  chain  of  events  ensures  that  the  con¬ 
trol  objective  is  achieved  (i.e.,  fhi(t)  — >  fh*  Vi  =  1,2, 3, 4), 
and  hence,  T  is  regulated  to  T* .  In  light  of  the  discussion 
above,  we  proceed  to  define  our  rotational  and  translation 
error  signals  whence  we  derive  the  respective  open-loop  er¬ 
ror  systems. 


R  =  ARA  1  Xh  =  Axh  n*  =  A  Tn*  .  (14) 

By  virtue  of  the  definitions  of  (13)  and  (14),  we  can  now 
write  (12)  as  follows 

Pi  =  ai(R+  xhn*T)  p*  =  aig33Gnp*  (15) 

where  Gn  (t)  =  G(t)/ff33(f)  (where  g33  (t)  /  0  V  i  by 
(9)  and  (15))  is  a  normalized  version  of  G(t).  From  the 
relationship  in  (15),  a  set  of  12  independent  linear  equa¬ 
tions  given  by  the  4  image  point  pairs  ( pi  (t)  ,p*)  (3  equa¬ 
tions  for  each  image  pair)  can  be  used  to  obtain  the  prod¬ 
uct  ai  ( t )  g33  (t)  and  the  matrix  Gn  ( t ).  Since  the  intrinsic 
camera  calibration  matrix  A  is  unknown,  standard  tech¬ 
niques  (e.g.,  [9],  [19])  that  rely  on  knowledge  of  the  Euclid¬ 
ean  homography  H  ( t )  cannot  be  applied  to  obtain  the 
translational  and  rotational  mismatch  between  T  and  T* . 
Rather,  we  employ  an  algorithm  based  on  vanishing  points 
to  obtain  R(t)  (details  available  upon  request).  Addition¬ 
ally,  the  subsequent  development  requires  knowledge  of  the 
depth  ratios  at  ( t )  Vz  =  1, 2, 3, 4  for  calculation  of  the  trans¬ 
lation  control  of  the  end-effector.  By  utilizing  the  relation¬ 
ship  in  (15),  a  nonlinear  system  of  equations  can  be  solved 
to  obtain  ai  ( t ).  Details  are  available  upon  request. 

IV.  Control  Objective  and  Open-loop  Error 
System 


A.  Rotation  Error  System 

Based  on  an  angle  axis  parameterization  of  the  rotation 
matrix  R  (t),  we  define  a  rotation  error-like  signal  ew  ( t )  £ 
5R3  as  follows  [15] 

ew  =  u9  (16) 

where  u(t)  £  5R3  represents  the  unit  axis  of  rotation  (i.e., 
the  eigenvector  corresponding  to  the  eigenvalue  equal  to 
one),  and  9  ( t )  denotes  the  rotation  angle  about  that  axis. 
As  stated  in  [18],  the  angle  axis  representation  of  (16)  is  not 
unique,  in  the  sense  that  a  rotation  of  —9  ( t )  about  —u(t)  is 
equal  to  a  rotation  of  9  ( t )  about  u(t).  A  particular  solution 
for  9  ( t )  and  u{t)  can  be  determined  as  follows  [18] 

9P  =  cos-1  (i  (tr  (R)  —  1)^  Wx  =  (17) 

where  the  notation  tr  (•)  denotes  the  trace  of  a  matrix,  and 
[•]x  denotes  the  following  skew-symmetric  matrix 


0 

_C3 

C2 

'  Ci  ' 

C3 

0 

-Cl 

VC  =  C2  ■ 

(18) 

— C2 

Cl 

0 

c3 

From  (17),  it  is  clear  that 

0  <  9P  (f)  <  7T.  (19) 


The  objective  of  this  paper  is  to  develop  a  visual  servo  The  constraint  in  (19)  is  consistent  with  the  computa- 

controller  in  order  to  ensure  that  the  robot  end-effector  is  in  tion  of  [u  (t)i  in  (17)  since  a  clockwise  rotation  (i.e., 

its  desired  final  position  and  orientation  (i.e.,  T  is  regulated  <  e  {t)  <  0)  is  equivalent  to  a  counterclockwise  rota- 
n-  There  are  6  degrees  of  freedom  (3  translational  tion  (Le.;  0  <  0  (t)  <  tt)  with  the  axis  of  rotation  reversed, 
and  3  rotational)  between  T  and  JF*;  hence,  our  control 

objective  is  satisfied  if  we  obtain  the  following:  R  (t)  —>  I3  2Any  point  Oi  on  n  can  be  utilized  in  the  subsequent  development; 

(where  the  notation  7;  denotes  an  1  x  1  identity  matrix)  and  the  image  point  Ou  and  hence,  the  subscript  1  is  utilized  in  lieu  of  i 

Xf  (t)  — »  0.  From  (4)  and  (5),  this  amounts  to  TYli(t)  — »  in  the  subsequent  development  of  the  translational  controller. 
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Based  on  (17)  and  the  functional  structure  of  the  object 
kinematics,  the  particular  solutions  9p  ( t )  and  up(t)  can  be 
used  in  lieu  of  9(t)  and  u(t)  without  loss  of  generality  and 
without  confining  9  ( t )  to  a  smaller  region.  Since,  we  do 
not  distinguish  between  rotations  that  are  off  by  multiples 
of  2n,  all  rotational  possibilities  are  considered  via  the  pa¬ 
rameterization  of  (16)  along  with  the  computation  of  (17). 

Based  on  the  previous  definitions,  we  can  now  develop 
the  open-loop  rotation  error  system.  Since  R  ( t )  is  unmea¬ 
surable  and  R  ( t )  can  be  calculated  via  a  vanishing  point 
algorithm  (details  available  upon  request),  it  is  desirable 
to  relate  the  parameterization  defined  in  (16)  to  a  set  of 
measurable  signals.  Based  on  (14),  we  know  that  R(t)  is 
similar  to  R(t),  and  hence,  R{t)  and  R(t)  have  the  same 
eigenvalues.  The  vector  u(t)  =  [fq  (t) ,  Ct2  (t) ,  M3  (t)]T  is 
the  computable  eigenvector  of  R  ( t )  corresponding  to  the 
eigenvalue  equal  to  one  and  is  related  to  the  unmeasurable 
u  ( t )  as  follows 

u  =  Au.  (20) 


where  ln(-)  denotes  the  natural  logarithm.  The  signals 
til  ( t ) ,  Vi  ( t ) ,  ul ,  and  can  be  measured  from  the  two  im¬ 
ages  directly.  By  exploiting  properties  of  the  ln(-),  the 
third  element  of  ev(t)  can  be  expressed  in  terms  of  oq(f). 
Since  ot\(t)  can  be  determined,  ev(t)  can  be  calculated. 
As  stated  earlier,  the  depth  ratio  oq  (t)  can  be  calculated; 
hence,  ev  ( t )  is  measurable. 

We  begin  by  observing  that  the  extended  image  coordi¬ 
nates  pe  ( t )  of  (25)  can  be  written  as  follows 


Cll  tt2  0 

-  Xi 

Zl 

a4 

Pe  = 

0  a3  0 

0  0  1 

m 

Zl 

.  ln(^i)  _ 

+ 

05 
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where  (6)  and  (9)-(ll)  were  utilized.  After  taking  the  time 
derivative  of  (24)  and  utilizing  the  relationship  in  (26),  we 
obtain 

ev  =  —Aemi  (27) 


To  facilitate  further  development,  we  define  an  auxiliary 
variable  9  ( t )  as  follows 


where  Ae  ( t )  £  ?R3x3  is  a  full-rank  matrix  defined  as  follows 


(28) 


=  cos 


ai 

a  2 

1 

C3 

(1  A 
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0 

a3 

a5  -  Vi 

(2  -1))  • 

(21) 

0 

0 
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Based  on  the  definition  of  R  (t)  in  (14)  and  the  properties 
of  tr  (■ ),  it  is  easy  to  see  that  tr  (J4)  =  tr  (R)\  hence,  we 
can  see  from  (17)  and  (21)  that  9  (t)  =  9  (t) .  Thus,  u(t) 
and  9  ( t )  are  measurable  signals  that  we  have  now  related 
to  u  ( t )  and  9  ( t ). 

The  open- loop  error  dynamics  for  eu(t)  can  be  expressed 
as  follows  (details  available  upon  request) 

where  u)c  ( t )  =  [ioc  1  ( t )  ,loC2  ( t )  ,uic 3  (t)]T  denotes  the  rota¬ 
tional  velocity  of  the  camera  expressed  in  the  coordinates 
of  T ',  Lu :  ( t )  £  ?R3x3  is  defined  as 


After  taking  the  time  derivative  of  (2),  mi(t)  can  be  ob¬ 
tained  as  follows  [15] 

mi  =  -vc  +  [mi]x  ljc  (29) 

where  vc  ( t )  =  [vci  ( t ) ,  vC2  ( t ) ,  vC3  ( t)]T  denotes  the  transla¬ 
tional  velocity  of  the  camera  expressed  in  the  coordinates 
of  T1  and  we  have  utilized  the  fact  that  [wc]x  =  —RRT. 
After  substituting  (29)  into  (27),  we  obtain 

ev  =  -—Aevc  +  Y(ioc,iii,vi)(j>  (30) 

where  Y  (•)  <j)  e  5R3  is  a  linear  parameterization  that  is  de¬ 
fined  as  follows 


Lu  —  I3  ~  ^  M  v  + 


1  - 


sine  (9) 


2;/ 


(23) 


and  u  ( t ) ,  9  ( t )  were  introduced  in  (17). 

B.  Translation  Error  System 

To  quantify  the  translation  mismatch  between  the  actual 
and  desired  end-effector  position,  we  define  the  translation 
error  ev  ( t )  =  [et,i  ( t ) ,  eV2  ( t ) ,  evz  ( t)]T  as  follows 


ev=pe-  p*e  (24) 

where  pe  ( t ) ,  p*e  G  5R3  denote  the  extended  image  coordi¬ 
nates  [15]  of  an  image  point  011  7r  in  terms  of  T  and  T* . 
respectively,  and  are  defined  as  follows 

pe=  [  ui  ui  ln(zi)  ]T  p*e  =  [  11*1  Ui  In  (2;* )  ]T 

(25) 


Y  (■)<!>  =  Ae  [mi]x  u>c 


(31) 


where  cj>  G  5Rm  is  the  unknown  constant  vector  of  camera 
calibration  parameters,  and  Y  (•)  £  5R3xm  is  a  measurable 
regression  matrix.  Based  on  (28)  and  the  definition  of  ai  ( t ) 
in  (8),  the  open-loop  error  dynamics  for  ev  ( t )  are  obtained 
by  rewriting  (30)  in  the  following  manner 


3 


evi  =  Yi  (vc2,vc3,ojc,  «i,  ui )<f>1  -  aivci 


X c 
a3 


ev2  —  Y2  ( VC3,UC ,  U l,Vi)  4> 2  —  alvc2 


(32) 


ZieV3  =  Y3  (uc,u i,vi)  <j) 3  -  aivC3 


where  T)  (•)  o;  Vi  ••=  1,2.3  are  linear  parameterizations  de- 
hned  as  follows 


v,  4  a2  («i  -  a4) 

Yi<f>i  =  —  [Y(t>\  -  ai  —vc2  -\ - 

ai  L  J 1  a\  ai 

Y2(j>2  =  —  [Y(i>\0  +  — — —aivc3 
«3  r  _  2  a3 

Y3<t>3  =  4  [Y<P\  3 


env  C3 


(33) 
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where  Y)  (•)  £  5Rlx<Ji  is  a  measurable  regression  matrix, 
0j  £  5R9i  are  unknown  constant  vectors,  and  [Y 0 ]  denotes 
the  element  of  the  vector  Y  (•)  0  of  (31). 

V.  Control  Development 
A .  Control  Design 

Before  we  present  our  camera  control  algorithm,  a  few 
definitions  need  to  be  introduced.  We  define  the  unknown 
constants  r]1,q2,‘q3  £  5R  as  follows 

b2 

Vi  =  r  V2  =  b4  V3  =  h  (34) 
03 


following  update  laws 

0i  =  riVf evi  02  =  V2Y2  ev2  03  =  r3y3T eu3 

(40) 

where  F,;  £  5R9iXl?i  \ji  —  1,2,3  are  positive  diagonal  gain 
matrices. 

B.  Stability  Analysis 

Theorem  1:  The  adaptive  update  laws  defined  in  (38) 
and  (40)  along  with  the  control  inputs  designed  in  (37) 
and  (39)  ensure  that  eu  ( t )  and  ev  (t)  are  asymptotically 
driven  to  zero  in  the  sense  that 


where  hi  £  5R  Vi  =  1, 2, ...,  5  are  unknown  constants  and  are 
defined  to  be  elements  of  A~T  in  the  following  sense 


&i  0  0 

b2  63  0 

64  65  1 


(35) 


Consistent  with  the  definition  of  A  in  (11)  and  the  strict 
positiveness  of  ai  and  a3  (see  (11)),  the  following  relation¬ 
ships  hold  true  for  each  bi 


lira  ||ew(t)|| ,  ||e„(t)||  =  0  .  (41) 

£—>00 

Proof:  To  analyze  the  stability  of  the  rotational  con¬ 
troller,  we  begin  by  defining  a  non-negative  function  de¬ 
noted  by  V\  (t)  £  5R  as  follows 

Vi  =  \eleu.  (42) 

After  taking  the  time  derivative  of  (42),  we  obtain 


h  =  —  >0  b2  =  ~  —  b3=  —  >0 

«i  aia3  a3 

a2a3  —  a3a3  (15 

64  — -  65  — - . 

a3a3  a3 


(36) 


Based  on  the  structure  of  the  open-loop  error  system  of 
(22),  the  constraints  on  the  availability  of  measurable  sig¬ 
nals,  and  the  subsequent  stability  analysis,  the  camera  an¬ 
gular  velocity  control  input  u>c  ( t )  is  designed  as  follows 


OJcl 

=  7l  0  Ml 

(37) 

o!C2 

=  72^  M2  -  Vl^cl 

OJ  c3 

=  l3~e  U3  -  f]2U>cl  -  fj3UJc 2 

where  9  (f),  u  (t)  have  been  previously  defined  in  (20)  and 
(21),  respectively.  In  (37),  7*  £  5R  Vi  =  1, 2, 3  denote  posi¬ 
tive  control  gains,  and  r),:  (t)  £  5ft  Vi  =  1, 2, 3  denotes  adap¬ 
tive  parameter  estimates  that  are  dynamically  generated 
as  follows 


Vi  =  ~b\9  tii <vci  -  b30  u2  (u>c2  +  r/jWd)  (43) 
—0u3  {u>c3  +  i]2i0ci  +  V3u,c2) 

where  we  have  utilized  the  relationships  of  (16),  (20),  (34), 
(35),  and  the  fact  that  e^Lu  =  ej.  The  motivation  for 
the  rotational  control  strategy  of  (37)  is  now  immediately 
obvious.  After  substituting  (37)  into  (43),  we  obtain 

Vi  =  —bi'fid2  u\  -  b3j292  ti22  -  7 392  u\  (44) 
-b30  u2ujcif]l  -  9  u3cocif]2  -  9  u3uic2fj3 

where  the  auxiliary  variables  fji(t)  £  5R  Vi  =  1,2,3  are 
defined  as 

Vi  =  Vi  ~Vi-  (45) 

In  order  to  take  care  of  the  last  three  sign  indefinite  terms  in 
(44),  we  augment  V\  (t)  by  defining  a  non- negative  function 
denoted  by  V0(t)  £  5R  as  follows 


Vi  =  -'J49u2ujci  r)2  =  —^f59u3ujci  f]3  =  -7  69u3ujc2 

(38) 

where  /yi  £  5R  Vi  =  4, 5, 6  denote  positive  update  gains. 

Motivated  by  the  open-loop  error  dynamics  for  ev  ( t )  of 
(32)  and  the  subsequent  stability  analysis,  the  translational 
control  vc  ( t )  is  designed  as  follows 


1 

ot\ 


Vd  =  —  (lrevi  +  Y\ 


(39) 


VC2  =  ^  (^8ev2  +  l202) 

VC3  =  ^  (79ew3  +  *303) 

where  7i  £  3?  Vi  =  7, 8, 9  denote  positive  control  gains. 
In  (39),  0j  (i)  £  Vi  =  1,2,3  denote  adaptive  parame¬ 
ter  estimate  vectors  that  are  dynamically  generated  by  the 


K  =  v,  +  wfi  +  w/2  +  wfi-  (46) 

After  taking  the  time  derivative  of  (46)  and  utilizing  (44) 
and  the  adaptive  update  laws  of  (38),  we  obtain 

K,  =  -&i7i<?2  ul-b37292  u22—y392  u\  <  -Xrf 2uTu  (47) 

where  we  have  utilized  the  fact  that  9  (t)  =9  (t),  and  the 
positive  constant  Ai  is  defined  as 


Ai  =  min  {6474,6372,73}  . 


(48) 


Given  the  relationship  in  (20)  and  the  fact  that  AT A  is 
positive  definite,  the  following  inequality  can  be  developed 


UTU  ^  Amin  (ATA)  UT’U 


(49) 
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where  Amjn  (•)  £  5R  denotes  the  minimum  eigenvalue  of  a 
matrix,  and  Amin  (At7L)  is  strictly  positive.  Based  on  (49), 
(47)  can  be  rewritten  as 

K  <  ~\2eleu  (50) 


where  the  strictly  positive  constant  A2  is  defined  as 

A2  =  AiAmin  (ATA)  •  ('51) 

From  (45),  (46),  and  (50),  we  can  now  show  that  eu  (t)  £ 
£2  PI  £00  and  that  77^,  ?)j  (t)  £  £00  Vi  =  1,  2,  3.  From  a  se¬ 
quential  inspection  of  the  relationships  of  (37)  and  (38),  we 
can  also  show  that  u>c(t),  r\i  (t)  £  £00  Vi  =  1,2,3.  Since 
|| u  (t) ||  =  1,  then  9  (t)  £  £2  fi  £00  from  our  preceding  as¬ 
sertions;  hence,  Lu  ( t )  £  £00  from  (23).  From  (22),  we  can 
state  that  eu(t)  £  £00  •  Hence,  we  can  apply  Barbalat’s 
Lemma  [17]  to  state  that  lim  eu  ( t )  ,6(t)  =  0. 

t—>  OO 

In  order  to  establish  the  asymptotic  regulation  of  the 
camera  translational  error,  we  define  a  non-negative  func¬ 
tion  denoted  by  14(f)  £  5R  as  follows 


V„  =  -  — 


zl 
a  1 
1  ~,T, 


e2  1  _  ( 

1,1  +  2U 


2  ,  x  *  2 
e«2  +  9^1et,3 


(52) 


+  20lFl 


-1 


^1  +  2^2  ^2  *02  +  2^3  £3  *03 


where  the  auxiliary  variables  ^  (f)  £  5ft  Vi  =  1,2,3  are 
defined  as 

0,-^0,  cv  (53) 

After  taking  the  time  derivative  of  (52)  and  utilizing  (32), 
(39),  and  (40),  the  following  expression  can  be  obtained 

Vv  =  -77eSi  -  7se?2  -  79^3  <  0.  (54) 


Based  on  (52),  (53),  and  (54),  it  is  easy  to  see  that  ev  (t)  £ 
£00  fl  £2  and  (f) ,  (t)  £  £oo.  From  the  definitions  of 

(24)  and  (25),  u\  ( t ) ,  v\  (t) ,  z\  (t)  £  £00  ■  From  the  positive¬ 
ness  constraint  placed  on  z* .  we  can  see  that  a,  (t)  £  £00  • 
From  (39),  we  can  see  that  vc  (t)  £  Given  the  preced¬ 
ing  assertions,  we  can  use  (32)  to  show  that  ev  (t)  £  £00- 
Since  ev  (t)  £  ^£2  and  ev  (t)  £  £oo,  Barbalat’s  Lemma 

[17]  can  be  applied  to  state  that  lim  ev  (t)  =  0.  Hence,  we 

t— >00 

have  proved  the  result  of  (41).  □ 

VI.  Conclusions 

In  this  paper,  a  kinematic  controller  is  developed  that 
asymptotically  forces  a  robot  end-effector  to  its  desired 
position  and  orientation  while  utilizing  a  camera  fixed  on 
the  robot  end-effector  for  visual  feedback.  A  homography- 
based  algorithm  that  utilizes  feature  point  recognition  on 
a  target  of  interest  as  well  as  points  on  the  plane  at  infinity 
are  utilized  in  conjunction  with  an  adaptive  camera  con¬ 
troller  in  order  to  obtain  the  result.  Since  the  controller 
is  designed  to  be  differentiable,  robot  dynamic  effects  can 
easily  be  incorporated  via  backstepping  design  methods  [7]. 
Future  research  will  target  the  development  of  adaptive  vi¬ 
sual  servo  controllers  for  the  fixed  camera  configuration. 


References 

[1]  F.  Chaumette  and  E.  Malis,  “2  1/2  D  Visual  Servoing:  A  Possi¬ 
ble  Solution  to  Improve  Image-Based  and  Position-Based  Visual 
Servoing,”  Proc.  of  the  IEEE  International  Conference  on  Ro¬ 
botics  and  Automation ,  San  Francisco,  California,  May  2000,  pp. 
630-635. 

[2]  F.  Chaumette,  E.  Malis,  and  S.  Boudet,  “2D  1/2  Visual  Servoing 
with  Respect  to  a  Planar  Object,”  Proc.  of  the  Workshop  on 
New  Trends  in  Image-Based  Robot  Servoing ,  1997,  pp.  45-52. 

[3]  J.  Chen,  A.  Behai,  D.  Dawson,  and  Y.  Fang,  “2.5D  Visual  Ser¬ 
voing  with  a  Fixed  Camera,”  Proc.  of  the  American  Control 
Conference ,  Denver  Colorado,  June  2003,  pp.  3442-3447. 

[4]  F.  Conticelli  and  B.  Allotta,  “Nonlinear  Controllability  and  Sta¬ 
bility  Analysis  of  Adaptive  Image-Based  Systems,”  IEEE  Trans¬ 
actions  on  Robotics  and  Automation ,  Vol.  17,  No.  2,  pp.  208-215 
(2001). 

[5]  P.  I.  Corke  and  S.  A.  Hutchinson,  “A  New  Hybrid  Image-Based 
Visual  Servo  Control  Scheme,”  Proc.  of  the  IEEE  Conference 
on  Decision  and  Control ,  Sydney,  Australia,  December  2000, 
pp.  2521-2527. 

[6]  K.  Deguchi,  “Optimal  Motion  Control  for  Image-Based  Visual 
Servoing  by  Decoupling  Translation  and  Rotation,”  Proc.  of  the 
Inti.  Conf.  on  Intelligent  Robots  and  Systems ,  B.C.  Canada,  pp. 
705-711,  Oct.  1998. 

[7]  Y.  Fang,  A.  Behai,  W.E.  Dixon,  and  D.  M.  Dawson,  “Adap¬ 
tive  2.5D  Visual  Servoing  of  Kinematically  Redundant  Robot 
Manipulators,”  Proc.  of  the  IEEE  Conference  on  Decision  and 
Control ,  Las  Vegas,  NV,  pp.  2860-2865,  December  2002. 

[8]  Y.  Fang,  W.  E.  Dixon,  D.  M.  Dawson,  and  J.  Chen,  “Robust 
2.5D  Visual  Servoing  for  Robot  Manipulators,”  Proc.  of  the 
IEEE  American  Control  Conference,  Denver,  Colorado,  June 
2003,  pp.  3311-3316. 

[9]  O.  Faugeras  and  F.  Lustman,  “Motion  and  structure  from  mo¬ 
tion  in  a  piecewise  planar  environment,”  International  Journal 
of  Pattern  Recognition  and  Artificial  Intelligence,  Vol.  2,  No.  3, 
pp.  485-508,  (1988). 

[10]  O.  Faugeras,  Three-Dimensional  Computer  Vision ,  The  MIT 
Press,  Cambridge,  Massachusetts,  2001. 

[11]  C.  A.  Felippa,  A  Systematic  Approach  to  the  Element- 
Independent  Corotational  Dynamics  of  Finite  Elements,  Cen¬ 
ter  for  Aerospace  Structures  Document  Number  CU-CAS-00-03, 
College  of  Engineering,  University  of  Colorado,  January  2000. 

[12]  G.  D.  Hager  and  S.  Hutchinson  (guest  editors),  Special  Section 
on  Vision-Based  Control  of  Robot  Manipulators,  IEEE  Trans. 
Robotics  and  Automation,  Vol.  12,  No.  5,  (1996). 

[13]  E.  Malis,  “Contributions  a  la  Modelisation  et  a  la  Commande 
en  Asservissement  Visuel,”  Ph.D.  Dissertation,  University  of 
Rennes  I,  IRISA,  France,  Nov.  1998. 

[14]  E.  Malis  and  F.  Chaumette,  “2  1/2  D  Visual  Servoing  with  Re¬ 
spect  to  Unknown  Objects  Through  a  New  Estimation  Scheme 
of  Camera  Displacement,”  International  Journal  of  Computer 
Vision,  Vol.  37,  No.  1,  pp.  79-97,  (2000). 

[15]  E.  Malis,  F.  Chaumette,  and  S.  Bodet,  “2  1/2  D  Visual  Servo¬ 
ing,”  IEEE  Transactions  on  Robotics  and  Automation ,  Vol.  15, 
No.  2,  pp.  238-250,  (1999). 

[16]  B.  Nelson  and  N.  Papanikolopoulos  (guest  editors),  Special  Issue 
of  Visual  Servoing,  IEEE  Robotics  and  Automation  Mag.,  Vol. 
5,  No.  4,  (1998). 

[17]  J.  J.  E.  Slotine  and  W.  Li,  Applied  Nonlinear  Control,  Prentice 
Hall,  Inc:  Englewood  Cliff,  New  Jersey,  1991. 

[18]  M.  W.  Spong  and  M.  Vidyasagar,  Robot  Dynamic  and  Control, 
John  Wiley  and  Sons,  Inc:  New  York,  New  York,  1989. 

[19]  Z.  Zhang  and  A.  R.  Hanson,  “Scaled  Euclidean  3D  Reconstruc¬ 
tion  Based  on  Externally  Uncalibrated  Cameras,”  IEEE  Symp. 
on  Computer  Vision,  pp.  37-42,  1995. 


5401 


